322 sequenced data split - #323
Open
Raindrop182 wants to merge 6 commits into
Open
Conversation
sklearn's train_test_split function raises an InvalidParameterError when test_size or val_size are zero.
Collaborator
Author
|
If we decide to add this feature, I can add a commit with unit tests. I have confirmed the code works, as I used it to split data for training already. |
tkswanson
marked this pull request as ready for review
July 29, 2026 17:35
tkswanson
reviewed
Jul 29, 2026
| val_size: float = 0.1, | ||
| test_size: float = 0.1, | ||
| seed: int = 42, | ||
| groupby_col: str = "sequence"): |
Member
There was a problem hiding this comment.
lets not set the default groupby_col, especially since train_test_val defaults to None. Instead let's suggest using sequence in the doc string
necessary to fix bug where if the manifest had a conf_col but was passed in with duplicate indices, this could potentially mess up the process of keeping only the highest confidence entry (ex: two rows with the same filepath, same indices, but different confidences would be kept instead of removed)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Edits
export_train_val_testto contain an optionalgroup_colparameter. Ifgroup_colis not None, a new helper function_stratified_grouped_datasplitis called, which returns train_df, val_df, and test_df such that every group ingroup_colbelongs to only one of test/train/val andlabel_colis stratified as best as possible. This allows for data splits that prevent data leakage by keeping images from the same sequence/camera location together.It does this by utilizing sklearn's
StratifiedGroupKFoldfunction. StratifiedGroupKFold splits data inton_splitsequal-sized sections, ensuring that each group ingroup_colbelongs to only one section, and thatlabel_colis stratified as best as possible. The helper function then grabs a certain number of sections to form the test and val datasets. (The logic is a bit unstraightforward because StratifiedGroupKFold's primary purpose is for k-fold cross validation)Some limitations:
StratifiedGroupKFoldsplits the data into an integern_splitssections, test_size and val_size are rounded to the nearest 0.05StratifiedGroupKFoldmakes sure that each group ingroup_colbelongs to only one of train/val/test. It tries to stratifylabel_colas best as possible, but it may not be mathematically possible for it to create a completely even distribution.export_train_val_testfunction also doesn't work for zero values since sklearn'strain_test_splitfunction raises anInvalidParameterErroriftest_sizeis 0.Closes #322